Skip to content

Conversation

@juanmleng
Copy link
Contributor

@juanmleng juanmleng commented Nov 21, 2025

Pull Request Description

What and why?

What

Pass test parameters through the LLM description generation pipeline so it can be included in the user prompt.

Why

Pass test parameters to the LLM description generation so that descriptions can reference parameter values (e.g., thresholds) that affect test results but aren’t included in the output tables. For example, tests like TooManyZeroValues use thresholds to determine pass/fail, but without access to the actual parameter value, the LLM assumes the default threshold. This can create inconsistencies between the pass/fail outcome and the results, and may lead to hallucinations due to misinterpreting the underlying data.

How to test

Run tests with parameters and check that parameters are part of the user message and that the generated description references them. This can be checked in Langfuse traces:

Screenshot 2025-11-21 at 15 09 00

What needs special review?

Dependencies, breaking changes, and deployment notes

https://github.com/validmind/backend/pull/2255

Release notes

Checklist

  • What and why
  • Screenshots or videos (Frontend)
  • How to test
  • What needs special review
  • Dependencies, breaking changes, and deployment notes
  • Labels applied
  • PR linked to Shortcut
  • Unit tests added (Backend)
  • Tested locally
  • Documentation updated (if required)
  • Environment variable additions/changes documented (if required)

@juanmleng juanmleng self-assigned this Nov 21, 2025
@juanmleng juanmleng added enhancement New feature or request internal Not to be externalized in the release notes labels Nov 21, 2025
Copy link
Contributor

@johnwalz97 johnwalz97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice 👍

@github-actions
Copy link
Contributor

PR Summary

This PR introduces functional changes to the testing description generation in the project. A new optional parameter, params, has been integrated into the functions in the test description module (specifically, generate_description, background_generate_description, and get_result_description). This allows for more customizable test behavior by passing a dictionary of parameters. In addition, the changes include minor documentation updates in the 'TooManyZeroValues' test to correct the default threshold representation from a percentage to a fraction representation (0.03% instead of 3%). The code in the test runner was also updated to pass the new params attribute, ensuring that these customizations are utilized during runtime. Overall, these enhancements aim to improve the flexibility and clarity of test descriptions without affecting existing functionality when the new parameter is not provided.

Test Suggestions

  • Add unit tests for generate_description, background_generate_description, and get_result_description to verify that the params argument is correctly integrated in the output.
  • Test the behavior when params is not provided to ensure backward compatibility.
  • Add integration tests for the run.py script to confirm that the params value from the result object is correctly passed to the description generation functions.
  • Validate the updated documentation in TooManyZeroValues to ensure the defaults are accurately reflected.

@juanmleng juanmleng merged commit 6e4d279 into main Nov 21, 2025
17 checks passed
@juanmleng juanmleng deleted the juan/sc-13271/pass-test-parameters-to-llm-test-result-description-generation branch November 21, 2025 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request internal Not to be externalized in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants